Multi-Scale Audio Spectrogram Transformer for Classroom Teaching Interaction Recognition

Classroom interactivity is one of the important metrics for assessing classrooms, and identifying classroom through image data limited by interference complex teaching scenarios. However, audio within are characterized significant student–teacher interaction. This study proposes a multi-scale spectrogram transformer (MAST) speech scene classification algorithm constructs interactive dataset to achieve teacher–student recognition in process. First, original signal sampled pre-processed generate multi-channel spectrogram, which enhances representation features compared with single-channel features; Second, order efficiently capture long-range global context globally modeled multi-head self-attention mechanism MAST, feature resolution reduced during extraction continuously enrich layer-level while reducing model complexity; Finally, further combination time-frequency enrichment module maps final output class map, enabling accurate category recognition. The experimental comparison MAST carried out on public environment self-built interaction datasets. Compared previous state-of-the-art methods datasets AudioSet ESC-50, its accuracy has been improved 3% 5%, respectively, reached 92.1%. These results demonstrate effectiveness field general smart domain.